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I. INTRODUCTION 


Properly managing program cost, schedule, and performance is critical to 
ensuring that programs meet their intended goals. A program manager is tasked with the 
responsibility of tracking planned progress versus actual progress to ensure that 
customers receive value for the time and money expended on programs. The earned 
value management (EVM) concept is a project management tool that integrates project 
scope with cost, schedule, and performance elements for optimum project planning and 
control. The Department of Defense (DoD) Earned Value Management Division states 
that earned value management should be fully embraced by the DoD acquisition 
community as an inherent part of the acquisition program management value chain 
because it provides program managers accurate and timely insight into cost, schedule and 
performance of DoD weapons systems and services programs (EVM, 2012). Today’s 
shrinking defense budgets focus DoD acquisitions on efficient use of time and money to 


give the best solution to the warfighter and the taxpayer. 


EVM has been used by DoD to forecast program cost and schedule growth over 
the last four decades. EVM can be defined as a project management technique that 
focuses on the planned worked accomplished and management’s planned budget for the 
work for the purpose of monitoring performance and predicting the final required costs 
and the time necessary to finish the project. EVM has been used by DoD since the 1960s 
as part of the Cost/Schedule Control Systems Criteria (C\SCSC) and more recently in 
32 EVMS guidelines for reporting program cost and schedule information. These 
guidelines have been adopted by the American National Standards Institute/Electronic 
Industries Alliance (ANSI/EIA-748 EVMS) and are being used by both government and 
industry. EVM tracks the cost and schedule progress and reports what the program truly 


earned for the budget that was spent. 


EVM is required on all defense acquisition program cost or incentive contracts 
over $20 million to help program managers track performance and predict potential 


overruns throughout the acquisition life cycle. In a revision to the DoD earned value 


management policy memorandum Michael Wynne, former Under Secretary of Defense 
Acquisition, Technology and Logistics (USD (AT&L), required the implementation of 
EVM methods and contract performance reports for all cost or incentive contracts valued 
at or greater than $20 million in then-year dollars and it was incorporated in Department 


of Defense Instruction (DoDI) 5000.02 (2008). 


EVM has been used by acquisition professionals to forecast the program’s 
schedule performance using cost-based indicators but does not completely address time- 
based indicators. Therefore, even though EVM has been used since the early 1960s as a 
program estimating tool it is viewed by some professionals as incomplete when 
predicting schedule performance indicators. An extension of EVM, called earned 
schedule, was introduced in 2003 as a tool to more accurately estimate schedule 
performance using EVM performance indicators that were lacking in traditional EVM 
metrics. 

Eamed schedule (ES) was developed by Walter H. Lipke and published in the 
March 2003 edition of the Program Management Institute College of Performance 
Management Journal (Lipke, 2003). It was developed to provide a unique method to 
determine schedule performance using the standard EVM performance indicators of 
budgeted cost of work scheduled (BCWS), actual cost of work performed (ACWP), 
earned value (EV), and budget at completion (BAC). The earned schedule concept 
identifies the time at which the amount of earned value (EV) accrued should have been 
earned and calculates schedule performance in terms of time, not costs. 

Earned schedule (ES) uses standard EVM performance indicator values and time- 
based equations to depict the schedule performance and may help a program manager 


estimate schedule performance more accurately. 
A. RESEARCH SCOPE 


This study captures earned value management and earned schedule data on four 
DoD MDAPs. It compared traditional EVM and ES performance metrics to forecast final 


program duration. 


The data collection included earned value data on four U.S. Army Chemical 
Materials Agency (CMA) programs. The programs were selected based on their 
performance measurement baselines and milestone similarities but due to the complexity 
of reporting period data only a specific portion of the data was selected for this research 
project. Specifically, data was collected from the work breakdown structure (WBS), 
operations phase, after each program was re-baselined to meet a Congressionally 
mandated April 2012 completion date. After program re-baseline, there were no further 
schedule changes to the performance measurement baseline (PMB) for the operations 
phase. Each program had operations phases that began on June 2007 and completed the 
incineration or neutralization of the chemical stockpile within a five-year period. The 
similarities between the operations phase, WBS and schedule requirements facilitated 


correlations between data analysis and conclusions across the programs. 


The PMB is highly important to earned value management because it provides the 
baseline plan to measure the program’s performance. It is the sum of the program’s 
planned cost over time and establishes the scope, schedule, and budget for a program. 
The earned value management implementation guide (EVMIG) describes a baseline as 
having the following characteristics: it accurately represents only authorized work on the 
contract, it includes a realistic network schedule baseline, and it includes a realistic time 
phased spread of budget/resources to the baselined schedule. Additionally, management 
makes a consistent commitment to enforce proper baseline change procedures and 
periodically review the remaining baseline to ensure that it remains executable (2006). A 
consistent schedule lent itself to comparable data between programs and was essential to 


answering the primary research question. 
B. RESEARCH OBJECTIVE 


The objective of this research project was to measure the accuracy of EVM and 
ES final duration forecast metrics on U.S. Army CMA programs. The traditional EVM 
method of estimating schedule performance uses cost-based indicators and Integrated 


Master Schedule (IMS) (DoDI 5000.02, 2008) to represent time. But, the research will 


not examine the DoD EVM practice of using the IMS to estimate program schedule. The 
research will only use derived mathematical equations to measure schedule performance 


metrics from EV data. 


The research will determine if the earned schedule method for final duration 
forecast produces a more accurate measure of the program’s final duration. The research 


paper will answer the following questions: 
1. Primary Research Question 


Is the ES method for the final duration forecast more accurate than EVM methods 


for final duration forecast for U.S. Army CMA programs? 
2. Secondary Research Questions 


a) How do EV and ES final duration forecast values compare? 


b) Can the ES final duration forecast be easily applied to DoD 
MDAPs? 


C. METHODOLOGY 


The research project methodology was developed to show the comparative value 
of EVM and ES final duration forecast methods to DoD programs. It collected traditional 
earned value management indicator data on four similar U.S. Army CMA programs and 
measured final duration forecast values for EVM and ES methods. The study used 
statistical analysis on the forecast values to determine the variation from the actual final 


duration over time and, thus, the accuracy of the methods. 


EVM data was collected on four U.S. Army CMA programs for work reported 
from October 2007 until the completion of the operations phase (munitions destruction). 
The earned value and planned value cost-based data was used to calculate EVM and ES 


performance metrics and produced forecasts for final program duration. 


A spreadsheet was used to calculate all necessary data for the research study. An 
ES spreadsheet for partial time periods was used as the basis for the data analysis and 
subsequently modified to include final duration forecast, mean squared error, and percent 


difference values. 


Final duration forecasts were determined using two EVM methods and one ES 
method. The three methods were the EVM planned value and earned value methods, and 
the earned schedule method. Respective forecasts for each reporting period, Mean 


Squared Error (MSE), and percent differences were calculated for each method. 


The mean squared error was selected because it reduced all forecast values to data 
that could be easily analyzed to answer the primary research question. It showed the 
statistical variation of the forecasted final duration about the actual duration value and 


was used to measure forecast method accuracy. 


This research project calculated MSE at program percent complete periods to 
measure the sensitivity of the methods to time. MSE was ranked over early (10-40 
percent), mid (40-70 percent), and late (70-100 percent) stages for each program. 
Smaller MSE values showed less spread in the data and therefore, forecasted values that 


were nearer to the actual final duration. 


This research project also used percent difference calculations to measure the 
accuracy of the final forecast value compared to the actual value over time. Percent 
difference values closer to zero represented forecasted values that were nearer to the 


actual final duration. 
D. ORGANIZATION 


This research paper is organized to allow for a logical progression from objective 
to conclusion and answering the primary research question: Is the ES method for the 
final duration forecast more accurate than EVM methods for final duration forecast for 


U.S. Army CMA programs? 


The INTRODUCTION chapter gives an expanded version of the research 
proposal. It establishes the scope, objective, and methodology of the research paper. 
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The BACKGROUND chapter introduces the concepts of earned value 
management and eamed schedule. It specifically addresses the key schedule performance 


metrics and applies them to the final duration forecasts. 


The CASE STUDY AND ANALYSIS chapter examines four U.S. Army CMA 
programs using earned value indicators over consistent schedule baselines. It documents 
the traditional EVM and ES forecasted final duration methods to measure their accuracy 
as predictors. An accuracy comparison is drawn between the methods to help answer the 


primary and secondary research questions. 


The SUMMARY AND RECOMMENDATIONS chapter provides the answer to 


the primary research question. 


Il. BACKGROUND 


A. EARNED VALUE MANAGEMENT CONCEPT 


In a memorandum for acquisition professionals, The Honorable Dr. Ashton Carter 
(former USD (AT&L)) emphasized the need to deliver better value to the taxpayer and 
warfighter by improving the way the Department of Defense does business. 

We must therefore abandon inefficient practices accumulated in a period 

of budget growth and learn to manage defense dollars in a manner this is, 

to quote Secretary Gates at his May 8, 2010 speech at the Eisenhower 


Library, “respectful of the American taxpayer at a time of economic and 
fiscal distress.” (Carter, 2010) 


As a means to achieve better value, cost and schedule metrics that track forecasted 
performance became key DoD program management tools. Earned value management is 


one such tool. 


Earned value management is a widely accepted practice for project management 
that is being used across the DoD, the Federal government, and industry. It is an 
integrated management system that coordinates the work scope, schedule, and cost goals 
of a program or contract, and objectively measures progress toward these goals. EVM 
provides a prediction of the final costs and schedule requirements and is used by program 


managers to: 


° quantify and measure program/contract performance 

e provide an early warning system for deviation from a baseline 

° identify and track risks associated with cost and schedule overruns 
° provide a means to forecast final cost and schedule outcomes 


Earned value simply allows the project manager to take advantage of their actual 
results and forecast the final results. If corrective actions are taken on overruns during 
the early stage of the program the program manager may change the forecasted outcome. 
EVM provides the program manager a powerful tool in the successful cost and schedule 


completion of a program. 


EVM uses performance indicators and metrics to objectively measure program 
progress against the budgeted plan. The performance indicators provide the input to 
measure the key schedule performance metrics used in EVM analysis. But, by 
themselves, earned value data indicators are not adequate to manage the project’s time 


dimension (Fleming & Koppelman, 2010). 
1. Performance Indicators 


Figure 1 shows notional data for a program with key EVM indicators plotted, 
namely, BCWS, ACWP, EV, and BAC. The EVM indicators are defined as follows: 


e BCWS, also called planned value (PV), is the ideal time-phased budget 
program plan for work currently scheduled or, more specifically, how 
much work the program should have accomplished at a specific time and 
cost. 


e ACWP is the actual cost incurred while accomplishing the program work. 
The ACWP value may be higher or lower than the BCWP at a specific 
time increment depending on the program’s progress. 


e EV is the budgeted cost for work performed (BCWP) on a program. It is 
the value of the completed work in terms of the work’s assigned budget, 
not simply what has been spent on the program. 


° BAC is budgeted cost at completion. 


A monthly contract performance report (CPR) is submitted to the government 
program manager with detailed performance indicator data (Data Item Description (DID) 


DI-MGMT-81466A, 2005) to monitor and measure EVM metrics. 


3500 5 


3000 + 


BAC 


2500 - 


2000 = 


--- EV 


Cost 


—_—_ N 
1500 - BCWS 


1000 | 


500 + 


Figure 1. EVM Indicators 


2. Schedule Performance Index 


An important schedule metric that can be derived from the EVM indicators is the 
program’s schedule performance index. The SPI helps predict the duration of the 
planned work by determining if the project is ahead, on, or behind schedule at any point 
in time on the BCWS line. It is calculated with cost-based data and can be an early 
indicator of program schedule problems. It is a measure of the program’s schedule 
efficiency and is calculated by taking the ratio of the earned value to the BCWS. The 


following equation was used to calculate SPI: 


The ratio yields a larger value when more work was performed than the planned 
value. Conversely, the ratio yields a smaller value when less work was performed than 
the planned value. Favorable, or schedule efficient programs have a SPI value greater 


than 1.0 and inefficient programs have a SPI value less than 1.0. 


Figure 2 shows notional SPI data for a program that completed behind schedule. 
The data represents a program that was scheduled for a 12-month completion date, but 
finished in 15 months. It shows baseline and actual SPI values plotted on the graph. 
Baseline SPI values show an ideal value throughout the program’s planned schedule and 
the SPI line shows the program’s actual schedule efficiency. The program’s SPI values 
fluctuate depending on the amount of planned work budgeted compared to the earned 


value of the work. 


The Figure 2 program had poor schedule performance and indicated that the 
program may not complete within the 12-month planned schedule. The SPI values were 
less than 1.0 until program completion and did not start to improve until month 10. At 
12 months, the program had not completed as planned but continued to improve until it 


completed at month 15. 


10 


1.05 - 


14 ~ 
a? 
¢ 
¢ 
¢ 
¢ 
¢ 
¢ 
4 
0.95 \— - 
‘ 
‘ 
‘ 
/ 
‘ 
tite ee ‘ 
a 09 - tor al ald rd 
a ‘Sennen, a --- SPI 
~~ , 
=© ¢ Ff 
‘ a — ideal 
‘ s 
> ¢ 
‘us? 
0.85 + 
0.8 + 
0.75 
1 2 3 4 5 6 7 8 9 10 41 12 13 14 45 
Time 
Figure 2. SPI for Late Completing Program 


Using traditional earned value management methods, actual SPI values will 
naturally reach an ideal value of 1.0 at program completion because SPI values are 
calculated with costs. EVM calculates schedule efficiency based on budgeted costs and 
therefore may not accurately reflect time-base schedule efficiency. At program 
completion all funds are expended and the SPI ratio always equals 1.0. Based on costs, at 
program completion all programs whether early, on time, or late, show a favorable 
schedule performance. Program managers must analyze the SPI trend and not just focus 
on the later program stage values because they may give unreliable indications of 


schedule performance when assessing risks and corrective actions. 
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3. Final Duration Forecast 


Another important metric related to schedule performance that can be derived 
from the EVM indicators is the independent estimate at completion (IEAC(t)) (Lipke, 
2009a). It predicts the program completion time based on elapsed time and forecast of 
work remaining on the program. The commonly applied form for IEAC(t) equals the 
elapsed time plus the forecast for work remaining. It can be expressed as the following 
formula: 


BAC — EV 


IEAC(t) = AT + 
WR 


where 
AT = actual time when PV and EV are reported 
WR = work rate factor that converts work into time. 


The two common work rates that are applied to forecast final duration are average 
BCWS (BCWS.,y) and average earned value (EVay). BCWSay value is used to depict 
program performance that is expected to progress according to plan. EVay is used to 
depict program performance that is expected to follow the current SPI trend. The two 
work rates are shown in the following equations and were used to calculate the 


IEAC(t)gcws and IEAC(t)gy values: 


Bows, = BOWS. 
n 
E 
EV,, _ Ve 
n 


where 
BCWS.um = Cumulative value of BCWS 
EV cum = Cumulative value of EV 


n = total number of time increments within AT. 
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EVM methods for forecasting project duration have been taught in training 
courses and used by project managers for four decades (Lipke, 2009a). But, EVM cost- 
based indicator values may not always accurately predict time. So, the earned schedule 
method was developed to measure schedule performance using standard EVM indicators 


and time. 
B. EARNED SCHEDULE CONCEPT 


In March 2003, Walter Lipke published a paper in The Measurable News that 
introduced an extension of earned value management that tracked program schedule in 
units of time rather than traditional EVM units of budget, called earned schedule (Lipke, 
2003). It was developed in response to the noted deficiency in using EVM cost-based 
indicators to effectively evaluate program schedule performance. Schedule performance 
is important because if a product is not delivered on time there can be serious 
repercussions. In addition to the likelihood of increased project costs, the customer, 
internal or external, is deprived of using the product, consequently preventing the 


delivery of their product or service (Lipke, 2009a). 


The basis for the earned schedule concept is straightforward. Identify the time 
at which the amount of earned value (EV) accrued should have been earned. By 
determining this time, time-based indicators can be formed to provide schedule variance 


and performance efficiency management information (Earned Schedule, 2012). 


ES also uses performance indicators and metrics to objectively measure program 
progress against the budgeted plan. The performance indicators provide the input to 
calculate the key schedule performance metrics used in ES analysis. ES can also provide 
schedule performance indicators that aide a program manager in the successful 


completion of a program. 
1. Performance Indicators 


From the time of the development of EVM indicators, it has been known that the 
schedule performance metrics are flawed and exhibit strange behavior over the final third 


of a project, when performance is poor (Lipke, 2003). As a program nears completion, 
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the BCWS and BCWP naturally approach each other so that all scheduled budgets have 
been used and the program theoretically completed on time using EVM schedule 
performance metrics. The standard EVM performance indicator data included in the 
CPR only represents budget data and any schedule metrics calculations from it will not 
measure time. Therefore, EVM schedule performance metrics have not always been 


viewed by project mangers as being reliable indicators for schedule (Lipke, 2003). 


Figure 3 shows notional data for an acquisition program with EV and BCWS 
plotted on the graph. The new value, earned schedule, is calculated by projecting the EV 
data point onto the BCWS curve to determine where EV equals the planned value 
(BCWS) for the program. Graphically, the horizontal dashed line projection from the 
EV; curve to the BCWS curve identifies the time that amount of EV should have been 
earned in accordance with the schedule. Extending a vertical dashed line from the 


BCWS curve to the x- axis yields a time increment for the earned schedule value. 
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Figure 3. ES Indicators 
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The earned schedule principle can be graphically explained but an equation is 
needed to accurately calculate the ES values. The equation uses actual EV and BCWS 
data values that are supplied in monthly CPRs. The following equation uses linear 


interpolation to calculate the earned schedule: 


EV, 


t+2 


BCWS 


Es=t4| Swe: ). 


—BCWS, 


t+1 


The linear interpolation method that was used to determine ES is represented in 


Figure 4. 


BCWS t+2 


O Known points 


4 Point with unknown x axis 
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Figure 4. Linear Interpolation Method to Calculate ES (From Fast, 2012) 


Using t=8, EV,+2=2300, BCWS,=2135, BCWS;+1=2435 from Figure 3 data yields 
a value of 8.55 months, which indicates that the program was behind schedule because it 


took 10 months to accomplish work that was only planned for 8.55 months. 
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ES =8+ = = | =8.55 
2435-2135 
2. Schedule Performance Index 


The important performance metric that can be derived from the ES is the 
program’s schedule performance index, SPI(t). It is a representation of how efficiently a 
program is performing to schedule. Lipke derived it from the ratio between actual time 
expended (ATE) and earned schedule. The following equation was used to calculate 
SPI(t): 


SPI(t)=—— . 
ATE 


Just as in traditional EVM schedule performance metrics, an SPI(t) value greater 


than 1.0 is favorable and a value less than 1.0 is unfavorable. 


Figure 5 shows the notational SPI(t) data for an acquisition program that 
completed behind schedule. The data represents a program that was scheduled for a 12- 
month completion date, but finished in 15 months. Just as in traditional EVM, the 
baseline SPI values equaled 1.0 throughout the program’s planned schedule. But, the 
actual SPI(t) lines showed different values than the SPI graph from Figure 2. The 
program performed poorly throughout its’ duration and it was clear by the SPI(t) values 
that the program was in danger of not completing on time. All SPI(t) values are less than 
1.0 and after the planned 12-month completion date they did not improve as the program 
reached completion because the program completed behind schedule. The critical 


difference between EVM and ES schedule performance index metrics is time. 
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SPI(t) calculations factor in the time that the earned value should have been 
accomplished and the actual time it was accomplished. Therefore, SPI(t) does not 
necessarily reach favorable numbers for all programs because costs are not the primary 
variables. ES metrics yield values that vary depending on the schedule efficiency 


measured in time. 
3. Final Duration Forecast 


Just as with EVM, earned schedule performance metrics can also be used to 
forecast final duration. The major difference between the two methods is that ES 
incorporates time with the planned duration (PD) and SPI(t) values in the equation. 


Lipke derived the following equation to calculate ES final duration forecast TEAC(t)gs): 
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PD 


TEAC(t) 25 ~ SPI(t) . 


A simulation study published in The Measurable News journal compared EVM 
and ES duration forecast methods and concluded that the ES metrics outperform, on the 
average, EVM duration forecast methods. Also, the ES method is more reliable in all 


stages (early stage, mid stage, late stage) of the project life cycle (Vanhouke, 2007). 
C. STATISTICAL ANALYSIS 


Statistical methods were used to determine the difference between estimated 
values (forecasted final duration) and true values (actual final duration) and indicate the 
accuracy of the forecast. Lipke used the following equation, termed standard deviation, 
to determine the relative accuracy of the forecasted value about the actual value in his 


research (Lipke, 2009b): 


> UEAC(t),, (i) -— FD)” 
Standard Deviation = 4/— 


n-l 


Lipke’s equation attempted to determine the variance of the forecasted values 
about the actual value at each time increment. But, a major difference between Lipke’s 
equation and the common standard deviation equation is a mean value. Standard 
deviation calculates deviation about a mean value and Lipke’s equation does not use a 
mean value, rather, it uses the fixed final duration value. This variation from the 
common standard deviation equation led the author to examine an alternate statistical 


equation for this paper. 


The author chose the mean squared error equation to measure the variance in the 


forecasted value about the actual value: 


MSE = =! (IEAC(t), - FD))’ 
n 


i=1 
where 


IEAC(t); = independent estimate at completion for each reporting period 
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FD = actual final duration 
n = number of reporting periods (entire known population). 


The MSE and Lipke’s standard deviation equation are very similar to each other. 
Both measure the squared value of the summation of the difference between IEAC(t) and 
FD divided by a number of time increments. In Lipke’s equation, the square root of the 
summation squared is shown to equal the standard deviation. Statistically, the squared 
root of the variance equals the standard deviation. The author chose MSE to show the 
variance in the forecast about the actual value but, taking the square root of MSE yields 
the root mean square deviation (RMSD), also known as standard deviation. Therefore, 
the RMSD is analogous to Lipke’s standard deviation equation and the author believes 
that MSE accurately measures the error between the forecasted and actual value while 


still maintaining the basis of Lipke’s research. 


A lower MSE indicated that the forecasted duration was closer to the actual 
duration, whereas a higher MSE indicated that the forecasted duration was further from 
the actual duration. The goal of this equation is to show an IEAC(t) method that is 


closest to the actual duration. 


The author also used the percent difference equation to determine the difference 
in the forecasted value versus the actual value and measure forecasted value accuracy. 


Percent difference was calculated with the following equation: 


IEAC()-FD | 


Percent Difference = 100. 


The percent difference value must remain small enough throughout the program 
as to not negatively affect the schedule because too large a difference would produce a 


Nunn-McCurdy Act (1983, § 2433) significant schedule breach indication. 


Table 1 summarizes the statistical analysis results as applied to the forecasted 


final duration analysis. 
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Table 1. 


Interpretation of Statistical Analysis Values 


Mean Squared Error | Percent Difference IEAC(t) Accuracy 
Low Low High 
High High Low 


D. SUMMARY OF THE EVM AND ES PERFORMANCE METRICS 


The ES method for SPI(t) and final duration forecast showed advantages over the 
EVM method for SPI and final duration forecast. ES used time indicators in the schedule 
performance equations, rather than the EVM cost indicators, which may more clearly 
represent program time against the PMB. ES values were also easily measured with 
existing EVM performance indicator data and the schedule performance calculations only 


required known program times or dates. Table 2 shows a comparison between EVM and 


ES performance equations. 


Table 2. Summary of EVM and ES Performance Metric Equations 

Method SPI IEAC(t) 
E = 

VM EV AT + (BAC - EV) 

BCWS BCWS., 

AT + (BAC - EV) 

EV., 
ES ES PD 
ATE SPI(t) 
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Ill. CASE STUDY AND ANALYSIS 


A. INTRODUCTION 


In November 1985, Public Law 99-145, Section 1142 (50 United States Code, 
Section 1521) was introduced and required the safe destruction of the US military 
chemical weapons stockpile. It directed DoD to dispose of the lethal unitary chemical 
agents and munitions stored at eight Army installations and required disposal facilities to 


be cleaned, dismantled, and disposed of according to applicable laws and regulations. 


U.S. Army CMA was created to incorporate the former Program Manager for 
Chemical Demilitarization and portions of the U.S. Army Soldier and Biological 
Chemical Command into one agency. The agency’s headquarters, scientific, 
communications, and support staff are located at the Edgewood area of the Aberdeen 


Proving Ground, MD. 


The U.S. Army CMA is the world leader in programs to store, treat, and dispose 
of chemical weapons safely and effectively. The agency developed and used incineration 
and neutralization technologies to safely store and eliminate chemical weapons at seven 
stockpile sites while protecting the public, its workers and the environment. The U.S. 
Army CMA fulfilled its mission and safely destroyed the Nation’s aging chemical 


weapons using incineration and neutralization technologies. 


The U.S. Army CMA programs exceeded ten years from initial facility 
construction to complete facility closure and were funded in excess of $50 million each. 
The contracts were incentive type so they were required to follow DoD EVM policy in 


accordance with DoDI 5000.02 (2008). 


In a revision to the DoD EVM policy, Michael W. Wynne, former USD (AT&L), 
required programs valued at or greater than $50 million in then-year dollars follow the 
thirty-two management guidelines published in ANSI/EIA-748 EVMS, submit a monthly 
CPR, and a monthly IMS. 
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The four U.S. Army CMA programs selected for this research followed the DoD 
EVM policy requirements and provided timely and consistent EVM data to the program 
manager. The programs were selected for this research paper based on their common 
EVM data, WBS, and schedule characteristics. Specifically, data was collected from the 
WBS level for the operations phase (agent destruction) after each program was re- 
baselined to meet a Congress mandated April 2012 completion date. The operations were 


conducted in the same operations phase WBS level for each program. 


Each program had a single prime contractor that managed its validated EVMS 
with one or two subcontractors that provided EVM data to the prime contractor. The 
subcontractors were fully integrated into prime contractor so there was only one EVMS 


per program. 


Each program was re-baselined and it was a significant event. The prime 
contractors were offered large guaranteed monetary incentives to complete the operations 
phase within schedule. After the program re-baseline on June 30, 2007 there were no 
further changes to the schedule baseline for the operations phase and each was expected 
to complete by April 30, 2012. All four U.S. Army CMA programs had the same start 
date and same planned completion date, which facilitated correlations between data 


analysis and conclusions across the programs. 


Due to the sensitivity of the data no actual program names or contractors will be 
discussed in this research paper. The programs will be designated Program A, Program 
B, Program C, and Program D and all data and analysis will be anonymous. Further 
details on the programs and EV data must be coordinated through the U.S. Army CMA 


headquarters. 
B. PROGRAM A 


EVM data was collected for the operation phase (BAC = $207,289,000) over the 
reporting period 30 June 2007 to 30 September 2011 and used to measure schedule 


performance metrics. 
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Figure 6 shows the MSE of the forecasted final duration for the three different 
IEAC(t) methods. The MSE for each of the IEAC(t) methods converged near the actual 


duration throughout the program and are consistent with any type of forecast method. 


The ES MSE plot was significantly lower than the BCWS and EV plots 
throughout the program. The ES MSE plot confirms that the ES method gave a closer 
final duration forecast than the BCWS and EV methods. But, the plots only show that 
IEAC(t) ES was better relative to the BCWS and EV methods and does not indicate the 
true accuracy of the data. For example, during the early program stage the MSE values 
were at least 110 months squared for every reporting period. A MSE =110 translated into 


a deviation of at least 10 months from the actual duration. 


The percent of total program duration was calculated by dividing the forecasted 
deviation months with the total program duration months. The calculated value gave a 
clear indication of the relative accuracy of the forecasted value compared to the actual 
length (time) of the program. In the case of Program A, the forecasted value deviation 
was 20 percent of the actual program duration, which would not have been useful to a 
program manager for schedule planning because it would have signaled a false significant 
schedule breach indicator. Though lower than EV and BCWS methods, the IEAC(t) ES 
method would still not be helpful to a program manager because it showed Nunn- 


McCurdy breach indicators that were false. 
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Figure 6. Mean Squared Error Results for Final Duration Forecast—Program A 


Figure 7 shows the percent difference of the forecasted duration (IEAC(t)) from 
the actual duration for each of the three methods plotted over time. Again, all three 
methods converged near the actual final duration as time progresses because the closer 
the program gets to completion the more confidence the method had in predicting the 
actual duration. But, none of the IEAC(t) methods showed a high confidence in helping 
the program manager forecast events. All IEAC(t) method values were at least 
25 percent over the actual duration throughout the majority of the program. Specifically, 
forecast methods at 25 percent different from the actual duration translate into missing 
the end date by at least 13 months. That type of inaccuracy in the forecast values would 
immediately lead to a Nunn-McCurdy Act schedule breach indicator. But, history shows 
that after program re-baseline the program did not experience a breach and completed 


ahead of schedule. The data showed that acceptable values, less than 11 percent 
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difference to prevent a Nunn-McCurdy breach indicator, were generally not achieved 
until the very late stages of the program where it is easier to make accurate forecasts. 
Though ES IEAC(t) showed improvement over the BCWS and EV IEAC(t) methods it 


would still not give a program manager confidence in forecasting schedule performance. 
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Figure 7. Percent Difference Results for Final Duration Forecast—Program A 


C. PROGRAM B 


EVM data was collected for the operation phase (BAC = $377,521,000) over the 
reporting period 30 June 2007 to 29 January 2012 and used to measure schedule 
performance metrics. 

Figure 8 shows the MSE of the forecasted final duration for the three different 
IEAC(t) methods. The MSE for each of the IEAC(t) methods converged near the actual 


duration throughout the program and are consistent with any type of forecast method. 
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The ES MSE plot was significantly lower than the BCWS and EV plots 
throughout the early and mid program stages. The ES MSE plot confirms that the ES 
method gave a closer final duration forecast than the BCWS and EV methods. But, the 
plots only show that IEAC(t) ES was better relative to the BCWS and EV methods and 
does not indicate the true accuracy of the data. For example, during the early program 
stage the MSE values were at least 60 months squared for every reporting period. 
A MSE =60 translated into a deviation of at least 8 months from the actual duration. In 
the case of Program B, the forecasted value deviation was 15 percent of the actual 
program duration, which would not have been useful to a program manager for schedule 
planning because it would have signaled a false significant schedule breach indicator. 
Though lower than EV and BCWS methods, the IEAC(t) ES method would still not be 
helpful to a program manager because it showed Nunn-McCurdy breach indicators that 


were false. 
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Figure 8. Mean Squared Error Results for Final Duration Forecast—-Program B 
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Figure 9 shows the percent difference of the forecasted duration (IEAC(t)) from 
the actual duration for each of the three methods plotted over time. None of the IEAC(t) 
methods showed a high confidence in helping the program manager forecast events. All 
IEAC(t) method values were at least 15 percent over the actual duration throughout the 
majority of the program. Specifically, forecast methods at 15 percent different from the 
actual duration translate into missing the end date by at least 9 months. That type of 
inaccuracy in the forecast values would immediately lead to a Nunn-McCurdy Act 
schedule breach indicator. But, history shows that after program re-baseline the program 
did not experience a breach and completed ahead of schedule. The data showed that 
acceptable values, less than 10 percent difference, to prevent a Nunn-McCurdy breach 
indicator were generally not achieved until the very late stages of the program where it is 
easier to make accurate forecasts. Though ES IEAC(t) showed improvement over the 
BCWS and EV IEAC(t) methods would still not give a program manager confidence in 


forecasting schedule performance. 
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D. PROGRAM C 


EVM data was collected for the operation phase (BAC = $175,112,000) over the 
reporting period 30 June 2007 to 28 October 2011 and used to measure schedule 


performance metrics. 


Figure 10 shows the MSE of the forecasted final duration for the three different 
IEAC(t) methods. The MSE for each of the IEAC(t) methods converged near the actual 


duration throughout the program and are consistent with any type of forecast method. 


The ES MSE plot was significantly lower than the BCWS and EV plots 
throughout the program. The ES MSE plot confirms that the ES method gave a closer 
final duration forecast than the BCWS and EV methods. But, the plots only show that 


IEAC(t) ES was better relative to the BCWS and EV methods and does not indicate the 
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true accuracy of the data. For example, during the early program stage the MSE values 
were at least 57 months squared for every reporting period. A MSE =57 translated into a 
deviation of at least 8 months from the actual duration. In the case of Program C, the 
forecasted value deviation was 15 percent of the actual program duration, which would 
not have been useful to a program manager for schedule planning because it would have 
signaled a false significant schedule breach indicator. Though lower than EV and 
BCWS methods, the IEAC(t) ES method would still not be helpful to a program manager 


because it showed Nunn-McCurdy breach indicators that were false. 
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Figure 10. Mean Squared Error Results for Final Duration Forecast—Program C 


Figure 11 shows the percent difference of the forecasted duration (IEAC(t)) from 
the actual duration for each of the three methods plotted over time. Again, all three 


methods converged near the actual final duration as time progresses because the closer 
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the program gets to completion the more confidence the method had in predicting the 
actual duration. But, none of the IEAC(t) methods showed a high confidence in helping 
the program manager forecast events. All IEAC(t) method values were at least 
30 percent over the actual duration throughout the majority of the program. Specifically, 
forecast methods at 30 percent different from the actual duration translate into missing 
the end date by at least 16 months. That type of inaccuracy in the forecast values would 
immediately lead to a Nunn-McCurdy Act schedule breach indicator. But, history shows 
that after program re-baseline the program did not experience a breach and completed 
ahead of schedule. The data showed that acceptable values, less than 11 percent 
difference, to prevent a Nunn-McCurdy breach indicator were generally not achieved 
until the very late stages of the program where it is easier to make accurate forecasts. 
Though ES IEAC(t) showed improvement over the BCWS and EV IEAC(t) methods 


would still not give a program manager confidence in forecasting schedule performance. 


250 - 


vo 
ro 
s 
= —IEAC(t)ES 
B IEAC(t)BCWS 
. — -IEAC(t)EV 
60 
Months (total = 52) 
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E. PROGRAM D 


EVM data was collected for the operation phase (BAC = $113,232,000) over the 
reporting period 30 June 2007 to 26 November 2010 and used to measure schedule 


performance metrics. 


Figure 12 shows the MSE of the forecasted final duration for the three different 
IEAC(t) methods. The MSE for each of the IEAC(t) methods converged near the actual 


duration throughout the program and are consistent with any type of forecast method. 


The ES MSE plot was significantly lower than the BCWS and EV plots 
throughout the program. The ES MSE plot confirms that the ES method gave a closer 
final duration forecast than the BCWS and EV methods. But, the plots only show that 
IEAC(t) ES was better relative to the BCWS and EV methods and does not indicate the 
true accuracy of the data. For example, during the early program stage, the MSE values 
were at least 119 months squared for every reporting period. A MSE =119 translated into 
a deviation of at least 10 months from the actual duration. In the case of Program D, the 
forecasted value deviation was 24 percent of the actual program duration, which would 
not have been useful to a program manager for schedule planning because it would have 
signaled a false significant schedule breach indicator. Though lower than EV and BCWS 
methods, the IEAC(t) ES method would still not be helpful to a program manager 


because it showed Nunn-McCurdy breach indicators that were false. 
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Figure 12. Mean Squared Error Results for Final Duration Forecast—Program D 


Figure 13 shows the percent difference of the forecasted duration (IEAC(t)) from 
the actual duration for each of the three methods plotted over time. None of the IEAC(t) 
methods showed a high confidence in helping the program manager forecast events. All 
IEAC(t) method values were at least 30 percent over the actual duration throughout the 
majority of the program. Specifically, forecast methods at 30 percent different from the 
actual duration translate into missing the end date by at least 12 months. That type of 
inaccuracy in the forecast values would immediately lead to a Nunn-McCurdy Act 
schedule breach indicator. History shows that after program re-baseline the program did 
not experience a breach and completed ahead of schedule. The data showed that 
acceptable values, less than 14 percent difference, to prevent a Nunn-McCurdy breach 


indicator were generally not achieved until the very late stages of the program where it is 
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easier to make accurate forecasts. Though ES IEAC(t) showed improvement over the 
BCWS and EV IEAC(t) methods would still not give a program manager confidence in 


forecasting schedule performance. 
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Figure 13. Percent Difference Results for Final Duration Forecast—Program D 


F. U.S ARMY CMA CASE ANALYSIS SUMMARY 


In summary, analysis of the four U.S. Army CMA programs showed that the ES 
performance metrics were more accurate indicators of schedule performance than EVM 
metrics. But, they still did not forecast schedule accurately enough to significantly 
improve the program manager’s ability to effectively manage schedule. Each of the 
metrics showed IEAC(t) values that exceeded the DoD schedule threshold limits on 


MDAPs when in reality neither of the programs had a schedule breach during the 
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operations phase. The IEAC(t) methods only showed accurate values during the late 


stage program stages where forecast values are easier to predict. 


Table 3 shows a summary of ES IEAC(t) value accuracy compared to the Nunn- 


McCurdy Act significant breach threshold. 


Table 3. TEAC(t) ES Summary 


Program Total Months Significant Variation about FD 
aes MSE diff 
Threshold 
value | months | value | months 

(months) 

A 51 6 110 10 25 13 

B 55 6 60 8 15 9 

G 52 6 57 8 30 16 

D 41 6 119 10 30 12 
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IV. SUMMARY AND RECOMMENDATIONS 


A. SUMMARY 


This Joint Applied Project examined DoD EVM methods to predict schedule 
performance and final program duration along with the ES method to predict schedule 
performance and final program duration. This research project applied the schedule 
performance metrics to four U.S. Army CMA programs to determine if ES metrics were 


better schedule indicators than EV metrics. 


The research showed that the ES metrics were generally more accurate than the 
EV metrics throughout the program duration. The MSE and percent difference plots of 
the IEAC(t) compared to the actual duration showed significantly closer forecast values 
with the IEAC(t)gs method. But, the ES metrics were only more accurate indicators 
relative to the two EV metrics and did not forecast duration accurately enough to be a 
useful program management tool. ES metrics predicted false significant schedule breach 
indicators on each of the four programs, which would have incorrectly caused program 


stoppage, corrective actions, and potential program cancellation. 


The current DoD EVM policy requirement and the familiarity of EVM tools and 
techniques make the relatively new and unproven ES methods difficult to implement in 
DoD. But, this research showed that ES performance metrics, specifically the ES value, 
may have advantages for project management and control that may well prove beneficial 
for DoD programs. Recommendations are provided below on ways that DoD can use ES 


methods to improve schedule performance metrics for acquisition programs. 


B. RECOMMENDATIONS 


1. Recommendation #1 


Recommend program managers apply ES performance metrics to complement 


current EVMS metrics. 
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The ES metric was a more accurate predictor of schedule duration compared to 
traditional EV time metrics and it may be valuable to the program manager tool kit. It 
was easily measured from CPR EVM cost data and can help track schedule performance 
along with other schedule performance metrics. The author believes that the DoD 
acquisition workforce is sufficiently trained in EVM principles to effectively use EVM 


data, measure the ES metric, and skillfully interpret the results. 
2. Recommendation #2 


Recommend that ES performance metrics for IEAC(t) do not replace current DoD 


EVMS standards for schedule performance as a program management tool. 


ES metrics were not capable of forecasting the final duration within a sufficient 
level of confidence for a program manager. Each of the four U.S. Army CMA programs 
showed significant Nunn-McCurdy Act schedule breach indicators based on the IEAC(t) 
ES data that were false. A program manager using only ES performance metrics would 


have responded incorrectly to false forecast final duration values. 
3. Recommendation #3 


Recommend future research on the accuracy of ES performance methods 
compared to the EVM IMS (DI-MGMT-81650, 2005) and critical path method (CPM) 


schedule metrics. 


CPM is a program scheduling technique that aids in understanding the 
dependency of events in a program and the time required to complete them. It uses a 
logic network to predict final project duration by managing those sequences of activities 
necessary to complete program milestones. IMS is an integrated schedule that details the 
tasks necessary to ensure program execution. The IMS is traceable to the WBS and is 
used to measure progress toward meeting program objectives and integrate schedule 
activities with all related components. Both IMS and CPM were designed to track 
program schedule performance and a comparative study between them and ES IEAC(t) 


would show potential differences. 
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C. CONCLUSIONS 


In conclusion, this Joint Applied Project examined DoD EVM equations to 
predict final program duration and the ES method to predict final program duration. ES 
schedule performance metrics did not forecast final duration accurately enough to 
warrant using them as a program management tool for the four U.S. Army CMA 


programs. 


The author believes that some aspects of ES may be a program management tool 
that should be coupled with EVM methods to more completely integrate project scope 


with cost, schedule, and performance elements for optimum project planning and control. 
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